import os
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
EDA- Analysis¶
cf=pd.read_excel("D:\\Jigsaw\\Project-student attrition\\Student2.xlsx")
shape is a tuple that gives you an indication of the number of dimensions in the array. Here It gives number of Rows and columns
cf.shape
Pandas is a fantastic ecosystem of data-centric Python packages. Due to which is one of those packages and makes importing and analyzing data much easier.
head() method. Pandas head() method is used to return top n (5 by default) rows of a data frame or series.
cf.head()
The tail() function is used to get the last n rows. This function returns last n rows from the object based on position. It is useful for quickly verifying data, for example, after sorting or appending rows. Number of rows to select by default it will show last 5 rows in python.
cf.tail()
The info() function is used to print a concise summary of a DataFrame. This method prints information about a DataFrame including the index dtype and column dtypes, non-null values and memory usage. Inorder to print the full summary.
cf.info()
cf.columns
# It shows number of columns in entire data frame
cf.skew()
This shows some columns are highly skewed & needs attention Highly skewed column includes, STDNT_AGE & DISTANCE_FROM_HOME As well SECOND_TERM_EARNED_HRS is highly negatively skewed
#check percentiles for some columns
quan=[]
for i in list(range(1,101)):
quan.append(i/100)
print(cf['DISTANCE_FROM_HOME'].quantile(q=quan).tail())
print(cf['STDNT_AGE'].quantile(q=quan).tail())
#Normalise distance fomhome column to treat skewness.Normalization converts all data points to decimals between 0 and 1.
def normalize(column):
upper = column.max()
lower = column.min()
y = (column - lower)/(upper-lower)
return y
normalized = normalize(cf.DISTANCE_FROM_HOME)
normalized.describe()
cf.shape
#No of unique values per column
for column in cf.columns:
print(f"{column}: Number of unique values {cf[column].nunique()}")
print("==========================================================")
Categorical Features¶
demo1 = []
for column in cf.columns:
if 1 < cf[column].nunique() <= 2:
# if len(cf[column].unique()) == 2:
# if raw1[column].dtype == object and len(raw1[column].unique()) <= 20:
demo1.append(column)
print(f"{column} : {cf[column].unique()}")
print(cf[column].value_counts())
print("====================================")
demo1.remove('RETURNED_2ND_YR')
len(demo1)
demo2 = []
for column in cf.columns:
if 2 < cf[column].nunique() <= 20:
print(f"{column} : {cf[column].unique()}")
demo2.append(column)
print("====================================")
len(demo2)
demo3 = []
for column in cf.columns:
if 20 < cf[column].nunique() <= 100:
print(f"{column} : {cf[column].unique()}")
demo3.append(column)
print("====================================")
len(demo3)
demo4 = []
for column in cf.columns:
if cf[column].nunique() > 100:
print(f"{column} : {cf[column].unique()}")
demo4.append(column)
print("====================================")
len(demo4)
# It shows sum of missing values for each variables
cf.isnull().sum()
#It is group data with two different variables named under att
att=cf.groupby('RETURNED_2ND_YR')
# This shows the data type of that column
type('RETURNED_2ND_YR')
#it shows mean of all the variables wrt to 0 and 1
att.mean()
The describe() method is used for calculating some statistical data like percentile, mean and std of the numerical values of the Series or DataFrame. It analyzes both numeric and object series and also the DataFrame column sets of mixed data types
cf.describe()
# .value counts shows the counts of those variables like It shows number of Female and male students here
cf['STDNT_GENDER'].value_counts()
# It is used to group the data to see number of students left and returned
att=cf.groupby('RETURNED_2ND_YR')
att_count=cf.groupby('RETURNED_2ND_YR').count()
# it shows the number of left students under different categories
att_count
# It will plot the bar chart to see how many students left due to distane from home
import matplotlib.pyplot as plt
plt.bar(att_count.index.values,att_count['DISTANCE_FROM_HOME'])
plt.xlabel('Students left College')
plt.ylabel('Number of students')
plt.title ('DISTANCE_FROM_HOME')
plt.show()
# It gives count for number of students left and return
cf.RETURNED_2ND_YR.value_counts()
# Its new data frame to see all the students who left
cf_new=cf.query("RETURNED_2ND_YR=='0'")
# It shows first 10 left students data
cf_new.head(10)
# It shows missing values for new data frame for left students
cf_new.isnull().sum()
# It gives information of what are data types..int , float object
cf_new.info()
# It will put the missing values in descending order with their percentage for each variable
drp_col=pd.DataFrame(round((cf_new.isnull().sum()/cf_new.shape[0])*100,2).sort_values(ascending=False)).reset_index()
# It will show number of rows and columns for left std data frame
cf_new.shape
# It shows the percentage of missing values in descending orders
drp_col
## it will drop the coloumns first 20 rows, as these are having high number of misssing values, which will bias our analysis
c=drp_col.nlargest(19,0)['index'].tolist()
# It will show number of rows and columns
drp_col.shape
# It will create new data frame for selected data, here we have dropped columns which are above 25%
drop=drp_col[0:18]
It shows the data that we are not using for analysis We have dropped them as many are categorical & also not directly relevant for the analysis
drop
# new data frame created by name vp for drop by Name
vp=cf_new.drop(drop['index'], axis=1)
This data frame is only for the student who left college
# It shows the number of columns under data frame vp
vp.columns
vp.shape
# It shows Gender of left students
vp['STDNT_GENDER'].value_counts()
#profile report for the Entire Data
import pandas as pd,os,numpy as np,pandas_profiling as pf
pf.ProfileReport(cf)
# It will show null values for all columns in vp data frame
vp.isnull().sum()
Imputing missing values¶
# Replace the null values with mean value
vp['STDNT_TEST_ENTRANCE_COMB'].fillna(999,inplace=True)
#vp['CORE_COURSE_NAME_2_F'].fillna('ENG 1101',inplace=True)
vp['CORE_COURSE_NAME_2_F'].fillna(vp['CORE_COURSE_NAME_2_F'].mode().iloc[0],inplace=True)
#vp['CORE_COURSE_GRADE_2_F'].fillna('B',inplace=True)
vp['CORE_COURSE_GRADE_2_F'].fillna(vp['CORE_COURSE_GRADE_2_F'].mode().iloc[0],inplace=True)
#vp['CORE_COURSE_NAME_3_F'].fillna('ENG 1101',inplace=True)
#vp['CORE_COURSE_GRADE_3_F'].fillna('B',inplace=True)
# Replace the null values with mode value
vp['CORE_COURSE_NAME_3_F'].fillna(vp['CORE_COURSE_NAME_3_F'].mode().iloc[0],inplace=True)
vp['CORE_COURSE_GRADE_3_F'].fillna(vp['CORE_COURSE_GRADE_3_F'].mode().iloc[0],inplace=True)
# Replace the null values with mean value
vp['DISTANCE_FROM_HOME'].fillna(112,inplace=True)
#vp.drop(columns='CORE_COURSE_NAME_1_S',axis=1,inplace=True)
# Replace the null values with mean value
vp['HIGH_SCHL_GPA'].fillna(3,inplace=True)
# It will fill the null values with mean
# vp drop --it will drop the coloumn which is not relevant for our analysis
vp['FATHER_HI_EDU_CD'].fillna(4,inplace=True)
vp.drop(columns=['FATHER_HI_EDU_DESC'],axis=1,inplace=True)
# It will fill the null values with mean
# vp drop --it will drop the coloumn which is not relevant for our analysis
vp['MOTHER_HI_EDU_CD'].fillna(4,inplace=True)
vp.drop(columns=['MOTHER_HI_EDU_DESC'],axis=1,inplace=True)
# vp drop --it will drop the coloumn & we are dropping it as is not required for our analysis
vp.drop(columns='DEGREE_GROUP_DESC',axis=1,inplace=True)
#vp['CORE_COURSE_GRADE_1_F'].replace('INCOMPL','NOT REP',inplace=True)
# It will fill the null values with mean
# vp drop --it will drop the coloumn which is not relevant for our analysis
vp['DISTANCE_FROM_HOME'].fillna(112,inplace=True)
vp.drop(columns=['SECOND_TERM_ATTEMPT_HRS'],axis=1,inplace=True)
vp.isnull().sum()
vp.shape
Plotting the GRAPHS for all variables for further analysis¶
We are seeking the analysis for the Students who left after first year causing Attrition¶
# It shows Bar graph for Number of students left w.r.t Distance from home
import matplotlib.pyplot as plt
plt.style.use('ggplot')
plt.xlim(min(vp['DISTANCE_FROM_HOME']),200)
plt.hist(vp['DISTANCE_FROM_HOME'],bins='auto')
plt.title('DISTANCE_FROM_HOME')
plt.xlabel('DISTANCE_FROM_HOME')
plt.ylabel('No of student')
plt.show()
Above plot shows maximum number ofnstudents who left college are staying between 60 to 80 Km away from college
# It shows Bar graph for Number of students left w.r.t Distance from home
import matplotlib.pyplot as plt
plt.style.use('ggplot')
plt.xlim(min(cf['DISTANCE_FROM_HOME']),200)
plt.hist(cf['DISTANCE_FROM_HOME'],bins='auto')
cf[cf["RETURNED_2ND_YR"] == 0][column].hist(color='red',label='Attrited',alpha=1.0)
cf[cf["RETURNED_2ND_YR"] == 1][column].hist(color='orange',label='Retained',alpha=0.5)
plt.legend()
plt.title('DISTANCE_FROM_HOME')
plt.xlabel('DISTANCE_FROM_HOME')
plt.ylabel('No of student')
plt.show()
sns.boxplot(data=vp['DISTANCE_FROM_HOME'])
plt.title('DISTANCE_FROM_HOME')
plt.ylabel('No of student')
#check percentiles for some columns
quan=[]
for i in list(range(1,101)):
quan.append(i/100)
print(vp['DISTANCE_FROM_HOME'].quantile(q=quan).tail())
#Inorder to treat the outliers here & Since less than 1% values are there, we'll drop them for analysis
#raw[[raw["DISTANCE_FROM_HOME"]<=275 | raw["STDNT_AGE"]<=20]]
#raw1=raw.query("DISTANCE_FROM_HOME <= 275 & STDNT_AGE <= 20")
# UNMET_NEED--among students left
plt.xlim(min(vp['UNMET_NEED']),900000)
plt.hist(vp['UNMET_NEED'],bins=10)
plt.title('UNMET_NEED')
plt.xlabel('UNMET_NEED')
plt.ylabel('No of student')
plt.show()
# UNMET_NEED
plt.xlim(min(cf['UNMET_NEED']),900000)
plt.hist(cf['UNMET_NEED'],bins=10)
cf[cf["RETURNED_2ND_YR"] == 0][column].hist(bins=5,color='red',label='Attrited',alpha=1.0)
cf[cf["RETURNED_2ND_YR"] == 1][column].hist(bins=5,color='orange',label='Retained',alpha=0.5)
plt.legend()
plt.title('UNMET_NEED')
plt.xlabel('UNMET_NEED')
plt.ylabel('No of student')
plt.show()
sns.boxplot(data=cf['UNMET_NEED'])
plt.title('UNMET_NEED')
plt.ylabel('UNMET_NEED')
plt.style.use('ggplot')
plt.xlim(min(vp['EST_FAM_CONTRIBUTION']),800000)
plt.hist(vp['EST_FAM_CONTRIBUTION'],bins='auto')
plt.title('EST_FAM_CONTRIBUTION')
plt.xlabel('EST_FAM_CONTRIBUTION')
plt.ylabel('No of student')
plt.show()
plt.style.use('ggplot')
plt.xlim(min(cf['EST_FAM_CONTRIBUTION']),800000)
plt.hist(vp['EST_FAM_CONTRIBUTION'])
cf[cf["RETURNED_2ND_YR"] == 0][column].hist(bins=5,color='red',label='Attrited',alpha=1.0)
cf[cf["RETURNED_2ND_YR"] == 1][column].hist(bins=5,color='orange',label='Retained',alpha=0.5)
plt.legend()
plt.title('EST_FAM_CONTRIBUTION')
plt.xlabel('EST_FAM_CONTRIBUTION')
plt.ylabel('No of student')
plt.show()
Above plot shows Maximum students who left are because of low Family Contribution
sns.boxplot(data=vp['EST_FAM_CONTRIBUTION'])
plt.title('EST_FAM_CONTRIBUTION')
plt.ylabel('EST_FAM_CONTRIBUTION')
plt.style.use('ggplot')
plt.xlim(min(vp['COST_OF_ATTEND']),900000)
plt.hist(vp['COST_OF_ATTEND'],bins='auto')
plt.title('COST_OF_ATTEND')
plt.xlabel('COST_OF_ATTEND')
plt.ylabel('No of student')
plt.show()
plt.style.use('ggplot')
plt.xlim(min(cf['COST_OF_ATTEND']),900000)
plt.hist(cf['COST_OF_ATTEND'],bins='auto')
cf[cf["RETURNED_2ND_YR"] == 0][column].hist(bins=5,color='red',label='Attrited',alpha=1.0)
cf[cf["RETURNED_2ND_YR"] == 1][column].hist(bins=5,color='orange',label='Retained',alpha=0.5)
plt.legend()
plt.title('COST_OF_ATTEND')
plt.xlabel('COST_OF_ATTEND')
plt.ylabel('No of student')
plt.show()
sns.boxplot(data=vp['COST_OF_ATTEND'])
plt.title('COST_OF_ATTEND')
plt.ylabel('COST_OF_ATTEND')
plt.style.use('ggplot')
plt.xlim(min(vp['GROSS_FIN_NEED']),900000)
plt.hist(vp['GROSS_FIN_NEED'],bins='auto')
plt.title('GROSS_FIN_NEED')
plt.xlabel('GROSS_FIN_NEED')
plt.ylabel('No of student')
plt.show()
plt.style.use('ggplot')
plt.xlim(min(cf['GROSS_FIN_NEED']),900000)
plt.ylim(min(cf['GROSS_FIN_NEED']),3000)
plt.hist(cf['GROSS_FIN_NEED'],bins='auto')
cf[cf["GROSS_FIN_NEED"] == 0][column].hist(bins=5,color='red',label='Attrited',alpha=1.0)
cf[cf["GROSS_FIN_NEED"] == 1][column].hist(bins=5,color='orange',label='Retained',alpha=0.5)
plt.legend()
plt.title('GROSS_FIN_NEED')
plt.xlabel('GROSS_FIN_NEED')
plt.ylabel('No of student')
plt.show()
sns.boxplot(data=vp['GROSS_FIN_NEED'])
plt.title('GROSS_FIN_NEED')
plt.ylabel('GROSS_FIN_NEED')
plt.hist(cf['FATHER_HI_EDU_CD'])
plt.title('FATHER_HI_EDU_CD')
plt.xlabel('FATHER_HI_EDU_CD')
plt.ylabel('No of student')
cf[cf["RETURNED_2ND_YR"] == 0][column].hist(bins=5,color='red',label='Attrited',alpha=1.0)
cf[cf["RETURNED_2ND_YR"] == 1][column].hist(bins=5,color='orange',label='Retained',alpha=0.5)
plt.legend()
plt.show()
sns.boxplot(data=cf['FATHER_HI_EDU_CD'])
plt.title('FATHER_HI_EDU_CD')
plt.xlabel('FATHER_HI_EDU_CD')
plt.ylabel('No of student')
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
sns.set(style="ticks")
x = np.random.randn(100)
f, (ax_box, ax_hist) = plt.subplots(2, sharex=True,
gridspec_kw={"height_ratios": (.15, .85)})
sns.boxplot(data=vp['FATHER_HI_EDU_CD'])
sns.distplot(vp['FATHER_HI_EDU_CD'])
ax_box.set(yticks=[])
sns.despine(ax=ax_hist)
sns.despine(ax=ax_box, left=True)
plt.hist(vp['MOTHER_HI_EDU_CD'])
plt.title('MOTHER_HI_EDU_CD')
plt.xlabel('MOTHER_HI_EDU_CD')
plt.ylabel('No of student')
plt.show()
plt.hist(cf['MOTHER_HI_EDU_CD'])
plt.title('MOTHER_HI_EDU_CD')
plt.xlabel('MOTHER_HI_EDU_CD')
plt.ylabel('No of student')
cf[cf["RETURNED_2ND_YR"] == 0][column].hist(bins=5,color='red',label='Attrited',alpha=1.0)
cf[cf["RETURNED_2ND_YR"] == 1][column].hist(bins=5,color='orange',label='Retained',alpha=0.5)
plt.legend()
plt.show()
sns.boxplot(data=vp['MOTHER_HI_EDU_CD'])
plt.title('MOTHER_HI_EDU_CD')
plt.xlabel('MOTHER_HI_EDU_CD')
plt.ylabel('No of student')
plt.hist(cf['HIGH_SCHL_GPA'])
plt.title('HIGH_SCHL_GPA')
plt.xlabel('GPA score')
plt.ylabel('No of student')
cf[cf["RETURNED_2ND_YR"] == 0][column].hist(bins=5,color='red',label='Attrited',alpha=1.0)
cf[cf["RETURNED_2ND_YR"] == 1][column].hist(bins=5,color='orange',label='Retained',alpha=0.5)
plt.legend()
plt.show()
sns.boxplot(data=cf['HIGH_SCHL_GPA'])
vp.hist(figsize=(20,20))
plt.show()
LL=cf.columns.to_list()
len (LL)
plt.figure(figsize=(100,100))
for i, column in enumerate(LL,1):
plt.subplot(8,7,i)
cf[cf["RETURNED_2ND_YR"] == 0][column].hist(bins=25,color='red',label='Attrited',alpha=1.0)
cf[cf["RETURNED_2ND_YR"] == 1][column].hist(bins=25,color='orange',label='Retained',alpha=0.5)
plt.legend()
plt.xlabel(column)
plt.figure(figsize=(25,15))
for i, column in enumerate(demo1,1):
plt.subplot(2,2,i)
cf[cf["RETURNED_2ND_YR"] == 1][column].value_counts().plot.bar(color='Orange',label='Retained',alpha=0.5)
cf[cf["RETURNED_2ND_YR"] == 0][column].value_counts().plot.bar(color='red',label='Attrited',alpha=1)
plt.legend()
plt.xlabel(column)
plt.figure(figsize=(30, 30))
for i, column in enumerate(demo2, 1):
plt.subplot(6, 4, i)
cf[cf["RETURNED_2ND_YR"] == 1][column].hist(bins=10,color='orange',label='Retained',alpha=0.5)
cf[cf["RETURNED_2ND_YR"] == 0][column].hist(bins=10,color='red',label='Attrited',alpha=1.0)
plt.legend()
plt.xlabel(column)
plt.figure(figsize=(30, 30))
for i, column in enumerate(demo3, 1):
plt.subplot(6, 4, i)
cf[cf["RETURNED_2ND_YR"] == 1][column].hist(bins=10,color='orange',label='Retained',alpha=0.5)
cf[cf["RETURNED_2ND_YR"] == 0][column].hist(bins=10,color='red',label='Attrited',alpha=1)
plt.legend()
plt.xlabel(column)
Now we will see by Pandas profiling the total details of all the variables for Histogram, missing values, & the correlations That will help us to choose & identify the key Drivers It is a simple and fast way to perform exploratory data analysis of a Pandas Dataframe.
CORRELATIONS BETWEEN VARIABLES¶
col=cf.corr().nlargest(10,"RETURNED_2ND_YR").RETURNED_2ND_YR.index
sns.heatmap(cf[col].corr(), annot=True, cmap="RdYlGn", annot_kws={"size":10})
cf.drop("RETURNED_2ND_YR", axis=1).corrwith(cf.RETURNED_2ND_YR).plot(kind='barh', figsize=(10, 7))
#SECOND_TERM_EARNED_HRS & SECOND_TERM_ATTEMPT_HRS & HIGH School GPA are highly correlated
#Second term & First term are negative highly co-related.
#AGE & Student Identifier are co-related.
#Unmet needs & Distance From Home are negaticely co-related.
cf.isnull().sum()
cf.shape
cf.info()
cf.describe()
cf.columns.to_series().groupby(cf.dtypes).groups
vp_new1 = cf[["RETURNED_2ND_YR","GROSS_FIN_NEED","COST_OF_ATTEND","EST_FAM_CONTRIBUTION",
"DISTANCE_FROM_HOME","UNMET_NEED","STDNT_GENDER","HOUSING_STS","HIGH_SCHL_GPA","STDNT_MAJOR","STDNT_MINOR",'FATHER_HI_EDU_CD', 'MOTHER_HI_EDU_CD','IN_STATE_FLAG','FIRST_TERM_EARNED_HRS','SECOND_TERM_EARNED_HRS',]].copy()
New data frame created with critical factors which are significant for the student who left..And these critical factors are taken from the mean score with respect to Returned second year from histogram analysis, correlation matrix and pandas profiling..
vp_new1.head()
vp_new1['RETURNED_2ND_YR']=vp_new1['RETURNED_2ND_YR'].map(lambda x: 1 if x== 0 else 0)
This function used to convert 0 to 1 and 1 to 0, as python consider only 1 for modelling... and in our data 0 means not returned to second year
vp_new1.head()
vp_new1.shape
vp_new1.isnull().sum()
Machine Learning algorithms can typically only have numerical values as their predictor variables. Hence Label Encoding becomes necessary as they encode categorical labels with numerical values. To avoid introducing feature importance for categorical features with large numbers of unique values, we will use both Lable Encoding and One-Hot Encoding as shown below.
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
# Create a label encoder object
le = LabelEncoder()
# Label Encoding will be used for columns with 2 or less unique values
le_count = 0
for col in vp_new1.columns[1:]:
if vp_new1[col].dtype == 'object':
if len(list(vp_new1[col].unique())) <= 2:
le.fit(vp_new1[col])
vp_new1[col] = le.transform(vp_new1[col])
le_count += 1
print('{} columns were label encoded.'.format(le_count))
# convert rest of categorical variable into dummy
vp_new1 = pd.get_dummies(vp_new1, drop_first=True)
vp_new1.head()
Here lets check the missing values if any
vp_new1.isnull().sum()
Now Data cleaning by imputing missing values--Treating null values with their mean and mode respectively
vp_new1['HIGH_SCHL_GPA'].isnull().sum()
vp_new1['HIGH_SCHL_GPA'].fillna(3.2,inplace=True)
vp_new1['HIGH_SCHL_GPA'].isnull().sum()
vp_new1['FATHER_HI_EDU_CD'].isnull().sum()
vp_new1['FATHER_HI_EDU_CD'].fillna(2.56,inplace=True)
vp_new1['FATHER_HI_EDU_CD'].isnull().sum()
vp_new1['MOTHER_HI_EDU_CD'].isnull().sum()
vp_new1['MOTHER_HI_EDU_CD'].fillna(2.57,inplace=True)
vp_new1['MOTHER_HI_EDU_CD'].isnull().sum()
vp_new1['COST_OF_ATTEND'].isnull().sum()
vp_new1['DISTANCE_FROM_HOME'].fillna(112,inplace=True)
vp_new1['HIGH_SCHL_GPA'].fillna(3.2,inplace=True)
vp_new1['FIRST_TERM_EARNED_HRS'].isnull().sum()
vp_new1['SECOND_TERM_EARNED_HRS'].isnull().sum()
vp_new1['SECOND_TERM_EARNED_HRS'].fillna(vp_new1['SECOND_TERM_EARNED_HRS'].mean(),inplace=True)
vp_new1['HIGH_SCHL_GPA'].isnull().sum()
vp_new1.isnull().sum()
print('Size of Full Encoded Dataset: {}'. format(vp_new1.shape))
Splitting data into training and testing sets¶
Prior to implementating or applying any Machine Learning algorithms, we must decouple training and testing datasets from our master dataframe.
The data we use is usually split into training data and test data. The training set contains a known output and the model learns on this data in order to be generalized to other data later on. We have the test dataset (or subset) in order to test our model's prediction on this subset . Train/test is used to measure the accuracy of model
# assign the target to a new dataframe and convert it to a numerical feature
target = vp_new1['RETURNED_2ND_YR'].copy()
from sklearn.model_selection import train_test_split
vp_new1.head()
# let's use stratify=y to maintain the same ratio as in the training dataset when splitting the dataset
X_train, X_test, y_train, y_test = train_test_split(vp_new1,
target,
test_size=0.25,
random_state=7,
stratify=target)
print("Number transactions X_train dataset: ", X_train.shape)
print("Number transactions y_train dataset: ", y_train.shape)
print("Number transactions X_test dataset: ", X_test.shape)
print("Number transactions y_test dataset: ", y_test.shape)
X_train.head()
X=vp_new1.drop("RETURNED_2ND_YR",axis=1)
y=vp_new1["RETURNED_2ND_YR"]
X.head()
y.head()
# train and test are used to simulate the entire data..which will be used for building model for prediction
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.30,random_state=7)
X_train.head()
Building Machine Learning Models¶
Baseline Algorithms Let's first use a range of baseline algorithms (using out-of-the-box hyper-parameters) before we move on to more sophisticated solutions.
from sklearn.preprocessing import OneHotEncoder, LabelEncoder
from sklearn.model_selection import train_test_split
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import RandomizedSearchCV
from sklearn.model_selection import ShuffleSplit
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
import xgboost as XG
# Libraries for data modelling
from sklearn import svm, tree, linear_model, neighbors
from sklearn import naive_bayes, ensemble, discriminant_analysis, gaussian_process
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from xgboost import XGBClassifier
from sklearn.ensemble import RandomForestClassifier
# Common sklearn Model Helpers
from sklearn import feature_selection
from sklearn import model_selection
from sklearn import metrics
# sklearn modules for performance metrics
from sklearn.metrics import confusion_matrix, classification_report, precision_recall_curve
from sklearn.metrics import auc, roc_auc_score, roc_curve, recall_score, log_loss
from sklearn.metrics import f1_score, accuracy_score, roc_auc_score, make_scorer
from sklearn.metrics import average_precision_score
## The algorithms considered in this section are: Logistic Regression, Random Forest, Decision Tree Classifier, XGboost¶
# selection of algorithms to consider and set performance measure
models = []
models.append(('Logistic Regression', LogisticRegression(solver='liblinear', random_state=7,
class_weight='balanced')))
models.append(('Random Forest', RandomForestClassifier(
n_estimators=100, random_state=7)))
models.append(('Decision Tree Classifier',
DecisionTreeClassifier(random_state=7)))
models.append(('xgboost',
XGBClassifier(random_state=7)))
acc_results = []
auc_results = []
names = []
# set table to table to populate with performance results
col = ['Algorithm', 'ROC AUC Mean', 'ROC AUC STD',
'Accuracy Mean', 'Accuracy STD']
cf_results = pd.DataFrame(columns=col)
i = 0
# evaluate each model using cross-validation
for name, model in models:
kfold = model_selection.KFold(
n_splits=10, random_state=7) # 10-fold cross-validation
cv_acc_results = model_selection.cross_val_score( # accuracy scoring
model, X_train, y_train, cv=kfold, scoring='accuracy')
cv_auc_results = model_selection.cross_val_score( # roc_auc scoring
model, X_train, y_train, cv=kfold, scoring='roc_auc')
acc_results.append(cv_acc_results)
auc_results.append(cv_auc_results)
names.append(name)
cf_results.loc[i] = [name,
round(cv_auc_results.mean()*100, 2),
round(cv_auc_results.std()*100, 2),
round(cv_acc_results.mean()*100, 2),
round(cv_acc_results.std()*100, 2)
]
i += 1
cf_results.sort_values(by=['ROC AUC Mean'], ascending=False)
Classification Accuracy is the number of correct predictions made as a ratio of all predictions made. It is the most common evaluation metric for classification problems. However, it is often misused as it is only really suitable when there are an equal number of observations in each class and all predictions and prediction errors are equally important. It is not the case in this project, so a different scoring metric may be more suitable.
Using Scikit-Learn’s RandomizedSearchCV method, we can define a grid of hyperparameter ranges, and randomly sample from the grid, performing K-Fold CV with each combination of values.
fig = plt.figure(figsize=(15,7))
fig.suptitle('Algorithm Accuracy Comparison')
ax = fig.add_subplot(111)
plt.boxplot(acc_results)
ax.set_xticklabels(names)
plt.show()
Above box plot for all modes shows the Accuracy comparison between each of them, Random forest and XGBOOST are at higher level, so selecting this 2 model for this case study for further analysis
fig = plt.figure(figsize=(15, 7))
fig.suptitle('Algorithm ROC AUC Comparison')
ax = fig.add_subplot(111)
plt.boxplot(auc_results)
ax.set_xticklabels(names)
plt.show()
Based on our ROC AUC comparison analysis, XGBoost and Random Forest show the highest mean AUC scores. We will shortlist these two algorithms for further analysis.
Logistic Regression is a Machine Learning classification algorithm that is used to predict the probability of a categorical dependent variable. Logistic Regression is classification algorithm that is not as sophisticated as the ensemble methods or boosted decision trees method discussed below. Hence, it provides us with a good benchmark.
Random Forest is a popular and versatile machine learning method that is capable of solving both regression and classification. Random Forest is a brand of Ensemble learning, as it relies on an ensemble of decision trees. It aggregates Classification (or Regression) Trees. A decision tree is composed of a series of decisions that can be used to classify an observation in a dataset.
Random Forest fits a number of decision tree classifiers on various sub-samples of the dataset and use averaging to improve the predictive accuracy and control over-fitting. Random Forest can handle a large number of features, and is helpful for estimating which of your variables are important in the underlying data being modeled
XGBoost is an implementation of gradient boosted decision trees designed for speed and performance that is dominative competitive machine learning.¶
The two reasons to use XGBoost are also the two goals of the project: Execution Speed. Model Performance.
a) XGBoost Execution Speed Generally, XGBoost is fast. Really fast when compared to other implementations of gradient boosting.
b) XGBoost Model Performance XGBoost dominates structured or tabular datasets on classification and regression predictive modeling problems.
from imblearn.over_sampling import SMOTE
from collections import Counter
SS = SMOTE()
X, y = SS.fit_resample(X, y)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=7)
print(X.shape)
print(y.shape)
print(X_train.shape)
print(y_train.shape)
print(X_test.shape)
print(y_test.shape)
acc_results = []
auc_results = []
names = []
# set table to table to populate with performance results
col = ['Algorithm', 'ROC AUC Mean', 'ROC AUC STD',
'Accuracy Mean', 'Accuracy STD']
cf_results = pd.DataFrame(columns=col)
i = 0
# evaluate each model using cross-validation
for name, model in models:
kfold = model_selection.KFold(
n_splits=10, random_state=7) # 10-fold cross-validation
cv_acc_results = model_selection.cross_val_score( # accuracy scoring
model, X_train, y_train, cv=kfold, scoring='accuracy')
cv_auc_results = model_selection.cross_val_score( # roc_auc scoring
model, X_train, y_train, cv=kfold, scoring='roc_auc')
acc_results.append(cv_acc_results)
auc_results.append(cv_auc_results)
names.append(name)
cf_results.loc[i] = [name,
round(cv_auc_results.mean()*100, 2),
round(cv_auc_results.std()*100, 2),
round(cv_acc_results.mean()*100, 2),
round(cv_acc_results.std()*100, 2)
]
i += 1
cf_results.sort_values(by=['ROC AUC Mean'], ascending=False)
fig = plt.figure(figsize=(15,10))
fig.suptitle('Algorithm Accuracy Comparison')
ax = fig.add_subplot(111)
plt.boxplot(acc_results)
ax.set_xticklabels(names)
plt.show()
fig = plt.figure(figsize=(15, 7))
fig.suptitle('Algorithm ROC AUC Comparison')
ax = fig.add_subplot(111)
plt.boxplot(auc_results)
ax.set_xticklabels(names)
plt.show()
Random Forest Classifier¶
RF is much easier to tune than GBM. There are typically two parameters in RF: number of trees and number of features to be selected at each node. RF is harder to overfit than GBM.
This method is especially attractive for this application in the following cases: The real-world data is noisy and contains many missing values, some of the attributes are categorical, or semi-continuous. There are needs to integrate different data sources which face the issue of weighting them. We need high predictive accuracy for a high-dimensional problem with highly correlated features.
RF methods can handle a large amount of training data efficiently and are inherently suited for multi-class problems. Let's take a closer look at using the Random Forest algorithm. Fine-Tuning the Random Forest algorithm's hyper-parameters by cross-validation against the AUC score.
Let's take a closer look at using the Random Forest algorithm. we shall fine-tune the Random Forest algorithm's hyper-parameters by cross-validation against the AUC score.
As shown above, the results from GridSearchCV provided us with fine-tuned hyper-parameter using ROC_AUC as the scoring metric
Best score --0.92 shows the prediction accuracy for this model. It is achieved by performing Grid Search based on the estimators.
from sklearn.model_selection import GridSearchCV
param_grid = {"n_estimators":[50, 75, 100, 125, 150, 175]}
rf_classifier = RandomForestClassifier(random_state=7,max_features='sqrt',class_weight='balanced_subsample')
grid_obj = GridSearchCV(rf_classifier, param_grid,n_jobs=-1,scoring='roc_auc',cv=10)
grid_obj.fit(X_train, y_train)
grid_obj.best_score_
param_grid1 = {"max_depth":[5, 10, 15, 20, 25]}
grid_obj1 = GridSearchCV(rf_classifier, param_grid1,n_jobs=-1,scoring='roc_auc',cv=10)
grid_obj1.fit(X_train, y_train)
param_grid2 = {"min_samples_split":[2,4,6,8,10]}
grid_obj2 = GridSearchCV(rf_classifier, param_grid2,n_jobs=-1,scoring='roc_auc',cv=10)
grid_obj2.fit(X_train, y_train)
param_grid3 = {"min_samples_leaf": [1, 2, 3, 4]}
grid_obj3 = GridSearchCV(rf_classifier, param_grid3,n_jobs=-1,scoring='roc_auc',cv=10)
grid_obj3.fit(X_train, y_train)
plt.figure(figsize=(20,20))
fig,[[ax,ax1],[ax2,ax3]]=plt.subplots(2,2,figsize=(20,10))
ax3.plot(grid_obj3.cv_results_['mean_test_score']),
ax2.plot(grid_obj2.cv_results_['mean_test_score']),
ax1.plot(grid_obj1.cv_results_['mean_test_score']),
ax.plot(grid_obj.cv_results_['mean_test_score'])
ax.set_xlabel("n_estimators")
ax1.set_xlabel("max_depth")
ax2.set_xlabel("min_samples_split")
ax3.set_xlabel("min_samples_leaf")
print(grid_obj.best_params_)
print(grid_obj1.best_params_)
print(grid_obj2.best_params_)
print(grid_obj3.best_params_)
print(grid_obj.best_score_)
print(grid_obj1.best_score_)
print(grid_obj2.best_score_)
print(grid_obj3.best_score_)
grid_fit = RandomForestClassifier(random_state=7,max_features='sqrt',class_weight='balanced_subsample',
n_estimators= 175,
max_depth= 25,
min_samples_split= 2,
min_samples_leaf= 1)
grid_fit.fit(X_train, y_train)
importances = grid_fit.feature_importances_
indices = np.argsort(importances)[::-1] # Sort feature importances in descending order
#indices = np.argsort(importances)[0:9:-1] # Sort feature importances in descending order
names = [X_train.columns[i] for i in indices] # Rearrange feature names so they match the sorted feature importances
plt.figure(figsize=(40, 7))#.plt(kind='barh)'# Create plot
plt.title("Feature Importance") # Create plot title
plt.bar(range(X_train.shape[1]), importances[indices]) # Add bars
plt.xticks(range(X_train.shape[1]), names, rotation=90) # Add feature names as x-axis labels
plt.show() # Show plot
Random Forest helped us identify the Top 10 most important indicators (ranked in the table below) as: Second term earned hours, first term earned hours, Distance from home, Family contribution, cost of attend, Gross Financial needs Mother high edu CD Father High edu CD Students Gender Housing sts
EVALUATION¶
## Confusion Matrix
cnf_matrix = metrics.confusion_matrix(y_test, grid_fit.predict(X_test))
class_names=[0,1] # name of classes
fig, ax = plt.subplots()
tick_marks = np.arange(len(class_names))
plt.xticks(tick_marks, class_names)
plt.yticks(tick_marks, class_names)
# create heatmap
sns.heatmap(pd.DataFrame(cnf_matrix), annot=True, cmap="YlGnBu" ,fmt='g')
ax.xaxis.set_label_position("top")
plt.tight_layout()
plt.title('Confusion matrix', y=1.1)
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
# percentage of accuracy score approx=(731+669)/(731+74+133+669)
print('Accuracy of RandomForest Regression Classifier on test set: {:.2f}'.format(grid_fit.score(X_test, y_test)*100))
The Confusion matrix is telling us that we have 730 + 651 correct predictions and 75+152 incorrect predictions. In other words, an accurac of 85.94%.
Confusion Matrix: A confusion matrix is a summary of prediction results on a classification problem. The number of correct and incorrect predictions are summarized with count values and broken down by each class. This is the key to the confusion matrix. The confusion matrix shows the ways in which your classification model is confused when it makes predictions. It gives us insight not only into the errors being made by a classifier but more importantly the types of errors that are being made.
The confusion matrix is useful for giving you false positives and false negatives. The classification report tells you the accuracy of your model. Classification Rate/Accuracy: Accuracy = (TP + TN) / (TP + TN + FP + FN) = (732+673) /(673+ 732+ 73+ 129) = 0.88 Recall: Recall gives us an idea about when it’s actually yes, how often does it predict yes. Recall = TP / (TP + FN) = 673 (673 + 129) =0.839
Precision: Precsion tells us about when it predicts yes, how often is it correct. Precision = TP / (TP + FP)= 673/ (673+73) = 0.90
F-measure: Fmeasure = (2 Recall Precision) / (Recall + Presision) = 2 ( 0.84 0.90) / (0.839 + 0.90) = 2(0.756)/1.74=1.512/1.74=0.87
# Classification report for the optimised RF Regression
grid_fit.fit(X_train, y_train)
print(classification_report(y_test, grid_fit.predict(X_test)))
grid_fit.fit(X_train, y_train) # fit optimised model to the training data
probs = grid_fit.predict_proba(X_test) # predict probabilities
probs = probs[:, 1] # we will only keep probabilities associated with the employee leaving
grid_fit_roc_auc = roc_auc_score(y_test, probs) # calculate AUC score using test dataset
print('AUC score: %.3f' % grid_fit_roc_auc)
fpr, tpr, thresholds = roc_curve(y_test, grid_fit.predict_proba(X_test)[:,1])
plt.figure()
plt.plot(fpr, tpr, label='Model (Area = %0.2f)' % grid_fit_roc_auc)
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver Operating Characteristic')
plt.legend(loc="lower right")
plt.savefig('Random Forest_ROC')
plt.show()
Using probabilistic class prediction resulted in AUC score is: 0.926 which is higher than that best score during the optimisation step.
ROC Graphs AUC - ROC curve is a performance measurement for classification problem at various thresholds settings. ROC is a probability curve and AUC represents degree or measure of separability. It tells how much model is capable of distinguishing between classes. The green line represents the ROC curve of a purely random classifier; a good classifier stays as far away from that line as possible (toward the top-left corner).
XGBOOST¶
import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
from operator import itemgetter
import xgboost as xgb
import random
import time
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import average_precision_score
import matplotlib.pyplot as plt
from numpy import genfromtxt
import seaborn as sns
from sklearn import preprocessing
from sklearn.metrics import roc_curve, auc,recall_score,precision_score
import datetime as dt
import pandas as pd
from sklearn.model_selection import train_test_split
import xgboost as xgb
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import numpy as np
from sklearn.model_selection import RandomizedSearchCV
import time
from sklearn.metrics import confusion_matrix
from sklearn.metrics import classification_report
from sklearn.metrics import roc_auc_score
import matplotlib.pyplot as plt
from sklearn.metrics import roc_curve
from sklearn.metrics import average_precision_score
from sklearn.metrics import precision_recall_curve
steps = [('scaler', StandardScaler()), ('XGBoost', xgb.XGBClassifier())]
pipeline = Pipeline(steps)
parameters = {'XGBoost__booster': ['gbtree', 'gblinear', 'dart'],
'XGBoost__subsample': np.arange(0.5, 1, 0.05),
'XGBoost__max_depth': np.arange(3, 20, 1),
'XGBoost__colsample_bytree': np.arange(0.1, 1.05, 0.05),
'XGBoost__objective': ['reg:logistic','binary:logistic']}
model = RandomizedSearchCV(pipeline, parameters, n_iter=10, scoring='roc_auc', cv=4)
start_time = time.time()
model.fit(X_train, y_train)
elapsed_time = (time.time() - start_time)/60
print('Time to tune the model: {0:0.2f} min.'.format(elapsed_time))
print('Tuned Model Parameters: {}'.format(model.best_params_))
predictions = model.predict(X_test)
pred_prob = model.predict_proba(X_test)[:,1]
cm = pd.DataFrame(confusion_matrix(y_test, predictions))
cm['Total'] = np.sum(cm, axis=1)
cm = cm.append(np.sum(cm, axis=0), ignore_index=True)
cm.columns = ['Predicted No', 'Predicted Yes', 'Total']
cm = cm.set_index([['Actual No', 'Actual Yes', 'Total']])
print(cm)
print(classification_report(y_test, predictions))
logit_roc_auc = roc_auc_score(y_test, model.predict(X_test))
fpr, tpr, thresholds = roc_curve(y_test, model.predict_proba(X_test)[:,1])
plt.figure()
plt.plot(fpr, tpr, label='Model (Area = %0.2f)' % logit_roc_auc)
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver Operating Characteristic')
plt.legend(loc="lower right")
plt.savefig('XGBoost_ROC')
plt.show()
ROC Graphs AUC - ROC curve is a performance measurement for classification problem at various thresholds settings. ROC is a probability curve and AUC represents degree or measure of separability. It tells how much model is capable of distinguishing between classes. The green line represents the ROC curve of a purely random classifier; a good classifier stays as far away from that line as possible (toward the top-left corner). This is the worst situation. When AUC is approximately 0.5, model has no discrimination capacity to distinguish between positive class and negative class.
# Create ROC Graph
from sklearn.metrics import roc_curve
fpr, tpr, thresholds = roc_curve(y_test, model.predict_proba(X_test)[:,1])
rf_fpr, rf_tpr, rf_thresholds = roc_curve(y_test, grid_fit.predict_proba(X_test)[:,1])
plt.figure(figsize=(14, 6))
# Plot XGboost ROC
plt.plot(fpr, tpr, label='XGBoost (area = %0.2f)' % logit_roc_auc)
# Plot Random Forest ROC
plt.plot(rf_fpr, rf_tpr, label='Random Forest (area = %0.2f)' % grid_fit_roc_auc)
# Plot Base Rate ROC
plt.plot([0,1], [0,1],label='Base Rate' 'k--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Graph')
plt.legend(loc="lower right")
plt.show()
In Machine Learning, performance measurement is an essential task. So when it comes to a classification problem, we can count on an AUC - ROC Curve. When we need to check or visualize the performance of the multi - class classification problem, we use AUC (Area Under The Curve) ROC (Receiver Operating Characteristics) curve. It is one of the most important evaluation metrics for checking any classification model’s performance. It is also written as AUROC (Area Under the Receiver Operating Characteristics)
Area under ROC Curve (or AUC for short) is a performance metric for binary classification problems. The AUC represents a model’s ability to discriminate between positive and negative classes. An area of 1.0 represents a model that made all predictions perfectly. An area of 0.5 represents a model as good as random